The problem with this approach is that Hugging Face also supports remote URIs natively (e.g. https://huggingface.co/docs/datasets/en/package_reference/loading_methods#datasets.load_from_disk). I think we don't want to use fsspec in those cases, because Hugging Face very well could have a more efficient native path.

This is a hard problem to solve; we have similar concerns on the Ibis side (e.g. #1298), but intuitively I'd err on the side of leaving it up to the Hugging Face APIs.

See comment below on needing a filesystem.

In order to save a DatasetDict we have to be able to access it.

Hugging Face uses fsspec under the hood anyway 🤷

Does it have to have the extension, or is this too narrow? Will people come saying they have HDF5 files with .hdf5 extension instead of .h5?

Actually, look at the C4 example under https://huggingface.co/docs/datasets/en/loading#hugging-face-hub; there's an example with .json.gz extension.

Hmmm... maybe I'll just glob everything in the directory then?

It seems you can specify splits in the data_files argument of load_dataset (e.g. https://huggingface.co/docs/datasets/loading#json); would this be preferable to constructing the DatasetDict with multiple Dataset.from_* calls? I haven't looked into the implementation, but I'd be curious if there's a good reason to not use the higher-level API.

Referencing the same C4 example from above (https://huggingface.co/docs/datasets/en/loading#hugging-face-hub), it seems you can also pass the wildcard directly under data_files.

There's two things happening here:

• Loading a dataset from the Hub (handled by HFDataset now - its not perfect, but I'm ignoring it in this PR).
• Loading / saving from a filesystem (what I'm trying to implement here).

Loading

The load_dataset can handle all our needed types with a call like:

load_dataset("csv", data_files="path/to/data.csv")

Same goes for parquet, lance, hdf5, json.

Arrow datasets don't play nice like this though and use load_from_disk("path/to/arrow").

Saving

This procedure works for everything but Arrow:

data = Dataset.from_dict({"a": [1,2], "b": [3,4]})
data.to_parquet("data.parquet")
load_dataset("parquet", data_files="data.parquet")

I couldn't find a way to save a DatasetDict to a particular format.

This loop is what happens in the DatasetDict itself here:

https://github.com/huggingface/datasets/blob/4775eeba2d5e73349790f7575182d71d5cd8e1bf/src/datasets/dataset_dict.py#L1376-L1383

Which eventually makes it down to this private method that only will save to Arrow format.

This same goes for the individual Dataset objects. You have to use the to_<format> methods there's not a single method to rule them all like there is with loading. Lance and HDF5 don't actually have save methods handled by Hugging Face :(

Agree with your point on file name format strictness though.

How do you get an IterableDataset, for example? At the bottom of https://huggingface.co/docs/datasets/en/filesystems, it seems like you could do IterableDataset.from_dict(), but it's not clear you're ever doing that, right?

QQ: Do you need a data_files dict here to specify filepaths?

Sorry I see below it's being read from _load_args. Should this be read from save_args instead?

This is a really good point actually... it should be read from save_args.

However, the way its implemented right now would require a user to specify data_files in both the load and save args.

Concretely, it would have to look like this in yaml:

reviews:
  type: huggingface.CSVDataset
  path: data/01_raw/reviews
  load_args:
    data_files:
      labels: labels.csv
      data: data.csv
  save_args:
    data_files:
      labels: labels.csv
      data: data.csv

Which isn't the most user friendly... hmm...

I guess there are two options: swap the saving operation to read from save_args and require it to be specified twice or allow data_files as a top-level argument to the dataset.

In this second option,

reviews:
  type: huggingface.CSVDataset
  path: data/01_raw/reviews
  data_files:
    labels: labels.csv
    data: data.csv

would work and so would specifying data_files in both the save and load args, but something like:

reviews:
  type: huggingface.CSVDataset
  path: data/01_raw/reviews
  data_files:
    labels: labels.csv
    data: data.csv
  save_args:
    data_files:
      labels: labels.csv
      data: data.csv

would throw an error (regardless of if filenames match or not).

What do you think @ankatiyar? Or maybe there's another option you'd prefer.

If this information is used for both saving and loading, I don't mind if it's a top level key - you're right having to specify it for both load and save args might be tedious but maybe templating (with OmegaConfigLoader) could also help for that option and this approach can be documented -

_data_files:
  labels: labels.csv
  data: data.csv

reviews:
  type: huggingface.CSVDataset
  path: data/01_raw/reviews
  load_args:
    data_files: ${_data_files}
  save_args:
    data_files: ${_data_files}

I'm okay with either approach, just using load_args for saving stuff feels not quite right. WDYT? @ElenaKhaustova

+1 to @ankatiyar that using load_args for save behaviour feels off. I'd go with Option B (top-level data_files) rather than duplicating it across load_args/save_args:

• data_files describes the directory layout, which both load and save need to agree on — same shape as path, which is also top-level.
• Single source of truth, no risk of load_args and save_args drifting apart.
• Surfaces it in the constructor signature instead of hiding it inside an opaque load_args dict.
• OmegaConf templating works, but relies on users knowing the trick and still produces visually duplicated YAML.

While you're in this code, there's also a related bug worth fixing: _save_dataset_dict currently ignores the filename values in data_files on save — it synthesises f"{save_path}/{split}{ext}" regardless. So:

load_args:
  data_files:
    labels: my_labels.csv    # silently ignored on save
    data:   my_data.csv

…will save to labels.csv / data.csv but then load looks for my_labels.csv / my_data.csv → round-trip fails. The current validation only checks keys, not values, and the test fixtures all use {split}{ext} filenames so this never gets exercised.

So my suggestion:

1. Promote data_files to a top-level constructor arg.
2. Raise a clear error if it also appears in load_args or save_args.
3. Make the save loop use f"{save_path}/{data_files[split]}" so filenames round-trip.
4. Add a test with a custom filename (e.g. {"train": "my_train.csv"}) to lock the behaviour in.

Great! Agreed. I'll do this.

I know this is dumb, but can we include this in each block, in the spirit of blind consistency with how other datasets are. 😅

Just to confirm, this extension is only used if you don't specify save_data_files? Does this also work for loading e.g. .parquet.snappy or some other sorts of extensions?

I think so, but want to be sure.